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FOREWORD 


This  is  Volume  I  of  the  final  report  of  Task  Group  II  of  the  Weapon 
System  Effectiveness  Industry  Advisory  Committee  (WSEIAC).  It  is  sub¬ 
mitted  to  the  Commander,  AFSC  in  partial  fulfillment  of  Task  Group  II 
objectives  cited  in  the  committee  Charter.  The  final  report  is  contained  in 
three  separate  volumes: 

Volume  I  Contains  an  overview  of  Task  Group  II  findings, 
including  a  summary  of  Volumes  II  and  III,  conclusions, 
and  recommendations. 


Volume  II  contains  a  discussion  of  effectiveness  concepts, 
a  description  of  specific  tasks  required  to  evaluate  effec¬ 
tiveness,  and  a  detailed  example  illustrating  the  method. 

Volume  III  contains  descriptions  of  effectiveness  analysis 
methods  applied  to  four  typical  Air  Force  systems  using 
the  techniques  described  in  Volume  II. 

The  membership  of  Task  Group  II  was  as  follows: 


Mr.  D.  F.  Barber  (Chairman) 

Mr.  1.  Bosinoff 

Mr.  I.  Doshay 

Dr-  B.  J.  Flehinger 

Mr.  W.  Haigler 

Mr.  H.  J.  Kennedy 
Mr.  C.  R.  Knight  (Technical 
Director) 

Mr.  A.  J.  Monroe 
Mr  M.  H.  Saunders 
Mr.  I.  M.  Tall 
Mr.  H.  D.  Voegtlen 


RADC  (EMER) 

Sylvania  Electronics  System  Division 
Space  General  Corporation 
IBM  -  Thomas  J.  Watson  Research 
Laboratories 

Rocketdyne  -  Division  of  North 
American  Aviation,  Inc. 

ARINC  Research  Corporation 
ARINC  Research  Corporation 

TRW  Space  Technology  Laboratories 
OOAMA  (OONEW) 

Radio  Corporation  of  America 
Hughes  Aircraft  Company 


Other  task  group  reports  submitted  in  fulfillment  of  the  committee's 
objectives  are 

AFSC-TR-66-1  Final  Report  of  Task  Group  I 

"Requirements  Methodology" 

AFSC-TR-65-3  Final  Report  of  Task  Group  III 

"Data  Collection  and  Management  Reports" 
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AFSC-TR-65-4 

AFSC-TR-65-5 

AFSC-TR-65-6 


Final  Report  of  Task  Group  IV 
"Cost-Effectiveness  Optimization" 
Final  Report  of  Task  Group  V 
"Management  Systems" 

Final  Summary  Report 
"Chairman's  Final  Report" 


Publication  of  this  report  does  not  constitute  Air  Force  approval  of  the 
report's  findings  or  conclusions.  It  is  published  only  for  the  exchange  and 
stimulation  of  ideas. 
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In  order  that  this  report  of  Task  Group  II  may  be  studied  in  context  with 
the  entire  committee  effort,  the  purpose  and  task  group  objectives  as  stated 
in  the  WSEIAC  Charter  are  listed  below: 

Purpose 

The  purpose  of  the  Weapon  System  Effectiveness  Industry  Advisory 
Committee  is  to  provide  technical  guidance  and  assistance  to  AFSC  in  the 
development  of  a  technique  to  apprise  management  of  current  and  predicted 
weapon  system  effectiveness  at  all  phases  of  weapon  system  life. 

Task  Group  Objectives 

Task  Group  I  -  Review  present  procedures  being  used  to  establish  system 
effectiveness  requirements  and  recommend  a  method  for  arriving  at  require 
ments  that  are  mission  responsive. 

Task  Group  II-  Review  existing  documents  and  recommend  uniform  methods 
and  procedures  to  be  applied  in  predicting  and  measuring  systems  effective¬ 
ness  during  all  phases  of  a  weapon  system  program. 

Task  Group  III  -  Review  format  and  engineering  data  content  of  existing 
system  effectiveness  reports  and  recommend  uniform  procedures  for 
periodically  reporting  weapon  s/stem  status  to  assist  all  levels  of  manage¬ 
ment  in  arriving  at  program  decisions. 

Task  Group  IV  -  Develop  a  basic  set  of  instructions  and  procedures  for  con¬ 
ducting  an  analysis  for  system  optimization  considering  effectiveness,  time 
schedules  and  funding. 

Task  Group  V  -  Review  current  policies  and  procedures  of  other  Air  Force 
commands  and  develop  a  framework  for  standardizing  management  visi¬ 
bility  procedures  throughout  all  Air  Force  commands. 


IV 


ABSTRACT 


Concepts  of  system  effectiveness  measurement  and  prediction,  presented' in 
detail  in  Volume  II,  are  summarized  briefly  in  this  volume.  Eight  formal¬ 
ized  tasks  necessary  to  evaluate  effectiveness  are  reviewed.  Summaries  of 
four  illustrative  examples,  presented  in  detail  in  Volume  III,  are  given. 

,  These  examples  provide  useful  guidelines  for  effectiveness  evaluation  at 
various  phases  of  system  life  cycle.  Conclusions  concerning  the  present 
state  of  system  effectiveness  evaluation  are  presented.  A  series  of 
recommendations  are  proposed  for  Air  Force  adoption. 
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SECTION  I 


INTRODUCTION  AND  SUMMARY 


1.  Task  Group  II  Objective 

The  original  objective  of  Task  Group  II  is  summarized  as  follows: 

'Review  existing  documents  and  recommend  methods 
and  procedures  to  be  applied  in  predicting  and  measuring 
effectiveness  during  all  phases  of  a  program.  " 

2.  Products  of  the  Group 

In  fulfillment  of  the  above  objective,  Task  Group  II  prepared  a  series  of 
three  Volumes  described  below: 

Volume  I  is  a  summary  of  the  technical  content  of  the  re- 
maining  two  volumes  and  is  addressed  to  program 
management  personnel  and  higher.  It  contains  a  con¬ 
solidated  list  of  recommendations  and  conclusions. 

Volume  II  is  a  detailed  exposition,  by  example,  of  the 
recommended  mathematical  framework  in  which  it  is 
proposed  that  systerr  effectiveness  be  computed/measured. 

It  is  addressed  to  project  level  personnel  and  is  intended 
to  provide  a  working  knowledge  of  the  proposed  mathe¬ 
matical  methods. 

Volume  III  presents  detailed  analyses  of  four  Air  Force 
systems:  airborne  avionics,  ICBM  squadron,  radar 
surveillance,  and  spacecraft.  The  intent  of  the  examples 
is  to  provide  detailed  guidance  in  utilizing  the  proposed 
mathematical  framework. 
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SECTION  II 


DISCUSSION 


1.  Background 

The  high  cost  and  complexity  of  modern  military  systems  require  the 
most  efficient  management  possible  to  avoid  wasting  significant  resources 
on  inadequate  equipment. 

Efficient  systems  management  depends  on  the  successful  evaluation  and 
integration  of  numerous  different  but  interrelated  system  characteristics 
such  as  reliability,  maintainability,  perforn.ance  and  cost.  If  such  evalu¬ 
ation  and  integration  is  to  be  accomplished  in  a  scientific  rather  than 
intuitive  manner,  a  method  must  be  formulated  to  assess  quantitatively  the 
effects  of  each  system  characteristic  on  overall  system  effectiveness.  It  is 
such  a  method  which  Task  Group  II  of  WSEIAC  was  directed  to  prepare. 

2.  System  Life  Cycle  Considerations 

AFR  375-1  divides  the  life  cycle  of  a  system  into  the  Conceptual, 
Definition,  Acquisition  and  Operational  Phases.  The  resvilts  obtained  from 
the  evaluation  in  each  phase  will  differ  both  in  accuracy  and  in  intended  use. 

During  the  Conceptual  Phase,  the  predicted  effectiveness  of  various 
types  of  systems  proposed  to  meet  an  operational  need  will  be  a  critical 
factor  in  selecting  the  optimum  course  of  action  to  follow.  A  limited  amount 
of  data  will  be  available  at  this  phase,  so  effectiveness  predictions  will  be 
based  largely  on  experience  with  previous  systems. 

Effectiveness  models  generated  during  the  Definition  Phase  will  be  use¬ 
ful  in  determining  the  configuration  and  scope  required  of  a  system.  More 
accuracy  will  be  possible  than  was  inherent  in  the  Conceptual  Phase  as 
enough  detail  will  be  available  to  employ  such  techniques  as  "Prediction  by 
Function"  in  assigning  values  to  the  characteristics  comprising  system 
effectiveness. 

During  the  Acquisition  Phase,  systems  effectiveness  predictions  can  be 
employed  to  determine  the  inherent  capability  of  the  proposed  system  to 
achieve  its  required  effectiveness,  the  effects  of  proposed  changes  on 
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systems  effectiveness,  and  the  optimum  trade-off  of  system  characteristics. 
Because  much  more  will  be  known  of  the  system,  and  actual  tests  will  be 
performed,  the  accuracy  of  systems  effectivenes-s  predictions  will  be  much 
higher  than  in  the  preceding  phases 

Finally,  in  the  Operational  Phase,  it  will  still  be  necessary  to  employ 
prediction  techniques  to  evaluate  systems  effectiveness.  Actual  field 
measurements  will  be  performed,  but  these  will  apply  only  to  peacetime 
conditions.  These  measurements  must  be  utilized  in  a  systems  effective¬ 
ness  prediction,  model  to  indicate  the  capability  of  the  system  to  perform  its 
design  mission  in  a  state  of  war.  During  this  phase  the  prediction  will  be 
most  accurate  as  many  system  characteristics  may  be  measured.  Complete 
accuracy  will  never  be  possible  short  of  war,  as  such  essential  items  as 
enemy  strategy  will  not  be  completely  known.  A  highly  accurate  prediction 
of  systems  effectiveness  under  given  hypothetical  environments  will,  how¬ 
ever,  be  possible. 

3.  Review  of  Volume  II 

a.  Conc<  pts 

In  Volume  II,  system  effectiveness  has  been  defined  as  a  measure 
of  the  extent  to  which  a  system  may  be  expected  to  achieve  a  set  of  specific 
mission  requirements.  It  is  further  defined  to  be  a  function  of  the  system's 
availability,  dependability,  and  capability. 

Availability  is  a  measure  of  the  system  condition  at  the  start  of  the 
mission.  It  is  a  function  of  the  relationships  among  hardware,  personnel, 
and  procedures. 

Dependability  is  a  measure  of  the  system  condition  at  one  or  more 
points  during  the  mission,  given  the  system  condition{s)  at  the  start  of  the 
mission. 

Capability  is  a  measure  of  the  ability  of  a  system  to  achieve  the 
mission  objectives,  given  the  system  condition{s)  during  the  mission. 
Capability  specifically  accounts  for  the  performance  spectrum  of  a  system. 

The  usefulness  of  a  mathematical  representation  (model)  of  the 


above  factors  and  the  steps  essential  to  the  model's  construction  and  its  use 
in  system  evaluation  are  discussed  in  some  detail.  These  steps  include  the 
specific  tasks  which  must  be  undertaken  in  order  to  provide  the  necessary 
model  inputs.  These  also  require  the  analyst  to  itemize  the  conditions 
imposed  and  the  assumptions  made  in  performing  the  evaluation. 

b.  Task  Analysis 

System  effectiveness  evaluation/prediction  can  be  reduced  to  an 
ordered  set  of  tasks  as  follows: 

(1)  Mission  Definition 

.  Functional  description  of  purpose 
.  Quantitative  requirements 

(2)  System  Description 

.  General  configuration 
.  Block  diagram 
.  Time  line  analysis 

(3)  Specification  of  Figures  of  Merit 

(4)  Identification  of  Accountable  Factors 

.  Level  of  accountability 
.  Personnel  characteristics 
.  Procedure  characteristics 
.  Hardware  characteristics 
.  Logistics 
.  Data  constraints 

(5)  Model  Construction 

.  Assumptions 

.  Delineation  of  possible  mission  outcomes 
.  Delineation  of  significant  system  states 

(6)  Data  Acquisition 

.  Specification  of  data  elements 
.  Specification  of  test  methodology 
.  Specification  of  data  reporting  system 

(7)  Parameter  Estimation 


4 


(8)  Model  Exercise 

•  Numerical  evaluation  of  effectiveness  and  its  factors 

•  Comparative  system  analysis 

•  Parameter  variation  study 

The  examples  of  Volumes  II  and  III  adhere  rather  closely  to  this 
analysis  of  the  steps  required  to  achieve  a  system  effectiveness  evaluation/ 
prediction. 

c.  Analytical  Methods 

A  specific,  analytical  model  is  introduced  in  which  Effectiveness  is 
expressed  as  the  product  of  availability,  dependability,  and  capability.  In 
order  to  provide  for  the  treatment  of  the  variovxs  conditions  of  the  system, 
each  of  the  three  variables  is  expressed  as  a  vector  or  a  matrix.  This 
method  of  mathematical  representation  and  treatment  has  the  advantage  that 
it  methodically  considers  all  significant  states  of  the  system  and  their  con¬ 
tributions  to  mission  success,  and  permits  a  proper  consideration  to  be 
given  to  the  contributions  to  mission  accomplishment  of  "less-than-perfect" 
system  conditions. 

In  the  various  sections  of  Volume  II  dealing  with  the  determination 
of  the  vectors  and  matrices,  the  construction  and  application  of  sub-models 
is  discussed.  These  sub-models  are  required  to  determine  the  numerical 
values  of  the  elements  of  the  vectors  and  matrices.  While  several  typical 
sub-models  are  discussed  --  and  employed  in  sample  evaluations  --  it  was 
not  possible  in  the  time  allotted  to  show  all  sub -models  which  might  be 
employed  in  the  evaluation  of  any  system. 

In  highly  complex  systems,  realistic  assumptions  relating  to  the 
accountable  factors  often  make  the  analytical  formulation  of  Availability  (A), 
Dependability  (D),  and  Capability  (C)  matrices  impractical.  When  this  is 
the  case,  the  only  feasible  course  is  to  resort  to  either  analog  or  digital 
computer  simulation. 

Simulation  methods  available  are  so  numerous  and  varied  that  it  is 
impractical,  here,  to  give  a  preferred  method.  The  best  method  in  a  par¬ 
ticular  case  depends  on  the  nature  of  the  system,  the  phase  of  the  program. 
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and  the  precision  required.  For  example,  in  some  cases  it  may  be  desir¬ 
able  to  use  simulation  methods  only  to  provide  estimates  of  the  A,  D,  C 
matrix  elements;  in  other  cases  it  may  be  preferable  to  by-pass  the  inter¬ 
mediate  outputs  and  proceed  directly  to  an  overall  measure  of  effectiveness. 

Despite  the  possible  variations,  all  simulation  methods  for  esti¬ 
mating  effectiveness  have  some  fundamental  common  characteristics.  First, 
the  relations  between  accountable  factors  and  the  effectiveness  figure  of 
merit  must  be  mathematically  described;  second,  the  manner  in  which  the 
accountable  factors  may  vary  from  one  system  trial  to  another  must  be 
known  or  reasonable  assumptions  established;  and  third,  a  large  number  of 
repeated  system  trials  must  be  run  on  the  computer, using  randomly  selected 
values  from  the  statistical  distribution  of  the  accountable  factors,  to  ob¬ 
tain  the  resulting  system's  degree-of-success.  This  last  step  is  commonly 
referred  to  as  a  Monte  Carlo  procedure. 

Simulation  techniques,  like  analytical  methods,  can  be  used  to 
determine  the  sensitivity  of  the  system  figur e-of-merit  to  variations  in  the 
accountable  factors.  In  such  an  exercise,  deliberate  (rather  than  random) 
changes  are  introduced  to  the  expected  values  of  the  accountable  factors  or 
to  the  parameter  values  of  the  distributions,  and  the  Monte  Carlo  process 
is  repeated. 

In  Volume  II  the  use  of  simulation  techniques  is  illustrated  through 
application  to  a  specific  effectiveness  problem. 

4.  Review  of  Volume  III 

Effectiveness  evaluation  for  large  Air  Force  systems  is  a  complex  task, 
subject  to  many  variations  in  detailed  procedures  depending  upon  the  type  of 
system,  information  available  to  support  the  evaluation,  and  the  phase  of  the 
system  life  cycle  treated.  In  Volume  III  of  the  Task  Group  II  report,  a  pre¬ 
liminary  analysis  of  the  utility  of  the  evaluation  methods  proposed  has  been 
made,  using  several  systems  as  hypothetical  effectiveness  vehicles.  These 
exercises  could  not  reasonably  be  expected  to  surface  and  answer  all 
questions  that  might  occur  during  actual  evaluations;  however,  they  have 
indicated  some  typical  problems  and  suggested  solutions. 
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A  detailed  description  of  the  application  of  effectiveness  evaluation 
procedures  is  given  for  the  following  four  (4)  examples: 

(1)  The  avionics  system  in  a  tactical  fighter -bomber ; 

(2)  a  sqviadron  of  intercontinental  ballistic  missiles; 

(3)  a  fixed  radar  surveillance  and  threat  evaluation  system;  and 

(4)  a  spacecraft  syst^  n 

In  addition  to  the  aritty  of  system  types  included,  an  attempt  is  made 
to  illustrate  procedures  employed  at  different  phases  of  the  system  life 
cycle. 

Further,  each  example  is  intended  to  illustrate, to  a  different  level  of 
detail,  various  aspects  of  the  evaluation.  The  avionics  system  example,  for 
instance,  shows  the  possiblity  of  combining  independent  evaluations  of 
several  subsystems.  The  radar  example  shows  simplifications  which  can 
be  made  in  order  to  minimize  the  number  of  system  states  to  be  considered. 
In  the  ICBM  example,  illustrations  of  many  of  the  detailed  procedures  re¬ 
quired  to  evaluate  components  of  the  vectors  and  m?itrices  are  shown. 
Finally,  the  spacecraft  example  shows  techniques  for  determining  elements 
of  the  Dependability  matrix. 

These  examples  do  not  illustrate  all  possible  methods  of  application  and 
use  of  the  evaluation  procedures.  Rather,  they  are  intended  to  show 
applicable  methods  of  application,  areas  of  flexibility,  and  uses  which  might 
be  made  of  the  evaluations. 

a.  Airborne  Avionics  System  Example 

The  purpose  of  this  example  is  to  demonstrate  how  the  effective¬ 
ness  evaluation  techniques  proposed  by  Task  Group  II  may  be  applied  to  the 
avionics  system  of  a  tactical  fighter -bomber  aircraft.  The  example  con¬ 
siders  only  the  "bombing"  function.  Similar  analyses  could  be  made  for  its 
"fighter,  "  "ground  support,  "  etc.  ,  functions. 

It  is  assumed  that  the  effectiveness  evaluation  is  being  made  during 
the  Program  Definition  Phase  of  system  life.  Similar  evaluations  in  the  real 
world  would  also  be  necessary  for  system  configurations  established  during 
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the  Acquisition  and  Operational  Phases.  A  major  consideration  of  the  Pro¬ 
gram  Definition  Phase  is  "force  structure,"  i.  e.,  the  number  of  systems 
(aircraft)  required  to  accomplish  a  specific  mission.  The  example  illus¬ 
trates  how  the  results  of  the  effectiveness  evaluation  aid  in  making  trade¬ 
offs  and  ultimate  decisions. 

The  svstem  is  evaluated  as  a  tactical  weapon.  The  aircraft  is 
considered  to  be  deployed  at  an  advanced  base  in  the  theater  of  operations, 
and  is  called  upon  to  bomb  tactical  targets  in  enemy  territory.  It  is 
assumed  that  no  enemy  offensive  action  will  be  mounted  against  the  ad¬ 
vanced  base. 

At  any  random  time  when  an  execution  order  is  received,  the  air¬ 
craft  shall  take  off  immediately,  receive  a  target  assignment,  proceed  to 
target  area,  deliver  weapon  within  500  feet  of  target,  and  return  to  assigned 
operation  base. 

b.  ICBM  Fleet  Example 

It  is  the  specific  object  of  this  example  to  illustrate  the  analysis 
of  an  ICBM  fleet  in  terms  of  the  formal  mathematical  structure  adopted  by 
Task  Group  II  of  the  Weapon  System  Effectiveness  Industry  Advisory 
Committee.  In  particular,  the  analysis  illustrates  the  usefulness  of  models 
in  assessing  the  impact  of  potential  system  alterations. 

The  general  requirements  of  this  hypothetical  system  may  be 
stated  as  follows: 

Any  missile  of  the  ICBM  fleet  should  be  ready  to  accept  a  launch 
directive  at  a  random  point  in  time,  or  at  an  arbitrary  time  after  an  alarm 
condition  has  been  established  at  a  random  point  in  time.  It  should  then 
launch  successfully  within  a  prescribed  reaction  time,  fly  a  ballistic  tra¬ 
jectory,  penetrate  enemy  defenses,  arm,  fuse,  impact  within  the  prescribed 
target  area,  detonate  and  yield  as  planned  with  a  prescribed  probability  of 
target  destruction. 

Minimum  acceptable  and  objective  numerical  system  requirements 
for  availability,  countdown,  flight,  and  probability  of  kill  are  postulated  in 
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the  form  of  an  SOR. 


c.  Radar  Surveillance  System  Example 

This  example  illustrates  for  this  type  of  defense  sytem  the 
effectiveness  prediction  techniques  discussed  earlier  in  this  document.  The 
tasks  required  to  evaluate  system  effectiveness  have  been  considered 
throughout  the  four  phases  of  system  life,  and  the  increasing  amount  of 
detail  which  is  necessary  as  the  system  evolves  is  shown. 

The  requirements  of  this  system  are: 

(1)  Detect  airborne  objects  in  the  surveillance  sector 
at  a  range  of  not  less  than  3,  000  nautical  miles. 

(2)  Identify  the  objects,  and  determine  within  30 
minutes  whether  or  not  they  constitute  a  threat. 

d.  Spacecraft  System  Example 

This  example  illustrates  in  some  detail  a  method  by  which  Depend¬ 
ability  of  a  spacecraft  may  be  determined  from  conservative  estimates  of 
hardware  reliability.  This  approach  is  suggested  in  the  absence  of  large 
amounts  of  test  data  on  the  vehicle  being  evaluated.  This  usually  occurs 
during  the  Program  Definition  and  early  system  Acquisition  Phases  of  pro¬ 
grams  on  substantially  new  systems.  It  is  also  useful  for  evaluating 
extremely  costly  systems  of  which  only  a  few  are  to  be  constructed.  No 
effort  is  made  in  this  example  to  treat  availability  or  capability,  beyond 
illustrating  their  tie-in  with  Dependability  to  calculate  effectiveness. 

The  assumed  purpose  of  this  evaluation  is  the  determination  of 
critical  elements  in  the  proposed  spacecraft  configuration. 

The  spacecraft  system  shall  be  capable  of  placing  a  variety  of 
payloads,  including  multiple  satellites,  into  precise  orbits  about  the  earth. 
It  shall  have  the  capability  of  restarting  in  space  after  a  sufficient  coast 
period  dependent  on  the  specific  payload  and  attitude  orientation  in  space. 
The  system  shall  be  designed  as  an  upper  stage  rocket  propulsion  vehicle. 
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SECTION  III 


CONCLUSIONS 

1.  The  results  attained  by  Task  Group  II  constitute  a  highly  useful 
framework  for  system  effectiveness  decision  making  and  evaluation. 

The  examples  presented  in  the  report  shov/  that  it  is  possible  to  employ 
this  framework  in  a  variety  of  system  types.  However,  it  was  not 
possible  in  the  time  allotted  to  analyze  and  evaluate  all  existing  pro¬ 
cedures  for  treating  each  of  the  elements  involved  in  effectiveness. 
Therefore,  no  recommendations  could  be  made  for  specific,  detailed 
techniques  applicable  to  each  type  of  system. 

2.  System  effectiveness,  as  defined  by  Task  Group  II,  represents  a  com¬ 
bination  of  system  characteristics  of  major  importance  to  the  user. 

The  definition  proposed  is  felt  to  be  sufficiently  broad  to  be  applicable 
to  all  system  types  at  all  stages  of  system  development.  It  does,  how¬ 
ever,  restrict  consideration  to  the  ability  of  the  system  to  accomplish 
a  specified  mission.  It  does  not,  therefore,  include  such  factors  as 
cost  or  development  schedule.  These  factors,  and  their  trade-off  with 
effectiveness,  have  been  treated  by  Task  Group  IV. 

3.  System  effectiveness  is  susceptible  to  both  measurement  and  prediction. 
However,  because  it  is  generally  not  feasible  to  measure  a  system's 
effectiveness  under  the  actual  conditions  of  intended  use,  the  element 

of  prediction  is  always  contained  to  some  extent  in  the  general  effective¬ 
ness  evaluation. 

4.  A  general  mathematical  model  has  been  developed  which  expresses 
effectiveness  as  a  function  of  three  major  system  characteristics,  i.  e.. 
Availability,  Dependability,  and  Capability.  Each  of  these  character¬ 
istics  is  defined  so  that  the  basic  concepts  intended  to  be  treated  by  each 
are  expressed.  They  are  expressed,  however,  such  that  considerable 
flexibility  in  the  methods  of  treating  the  concepts  is  permitted.  The 
general  model  is  of  such  a  form  that  the  influence  on  mission  accomplish¬ 
ment  of  all  system  "states"  (modes  of  degraded  system  operation)  can 

be  considered. 
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The  eight-step  procedure  presented  outlines  a  series  of  tasks  which 
must  be  accomplished  in  order  to  evaluate  the  effectiveness  of  any 
system.  Again,  the  procedure  states  the  points  which  must  be  treated, 
yet  permits  considerable  flexibility  in  the  methods  of  treatment. 

Using  the  procedures  and  mathematical  model  developed  by  Task  Group 
II,  effectiveness  predictions  can  be  made  at  all  phases  of  the  system 
life  cycle.  In  the  Conceptual  Phase,  the  prediction  will  be  based  on 
experience  with  systems  of  similar  type,  and  should  be  contained  in  the 
SOR.  As  the  system  proceeds  through  the  Definition  Phase,  the  avail¬ 
ability  of  system  design  data  will  allow  predictions  to  be  made  on  the 
basis  of  equipment  or  subsystem  function.  During  the  Acquisition  and 
Operational  Phases,  data  derived  from  actual  equipment  or  subsystem 
exercise,  first  under  laboratory  conditions  and  later  in  the  field,  will 
permit  early  predictions  to  be  verified  and  modified  where  required  in 
the  light  of  more  current  experience. 

Validation  of  effectiveness  evaluation  techniques  should  be  accompJished 
on  one  or  more  AF  systems,  prior  to  implementation  of  the  evaluation 
program  contractually.  An  expansion  of  this  subject  will  be  given  in 
recommendations  to  follow. 

No  consideration  has  been  given  to  methods  and  procedures  for  effec¬ 
tiveness  demonstration.  Contractual  specification  of  a  quantitative 
effectiveness  requirement  must  be  accompanied  by  clearly  stated  test 
and  demonstratirn  procedures. 

Specification  of  an  effectiveness  number  in  contractual  documents  should 
include  specification  of  any  constraints  on  the  individual  effectiveness 
characteristics  (A,  D,  C)  resulting  from  operational  considerations. 
Contractual  trade-offs  would  be  permitted  only  within  these  constraints. 

Figure(s)-of-merit  developed  must  be  interpreted  in  light  of  data 
limitations  and  assumptions.  An  expression  of  the  analyst's  confidence 
in  the  results  should  be  given  along  with  the  numerical  values. 
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Step-‘by-3t<^^t)  techniques  for  using  effectiveness  evaluation  in  decision¬ 
making  are  required.  This  requirement  includes  such  subjects  as 
trade-off.?,  effects  of  corrective  actions  and  sensitivity  analyses. 

IZ.  Comprehensive  generic  data  sources,  especially  those  including  effects 
of -environmental  stresses,  are  currently  nonexistent.  This  is  the 
weakest  link  in  the  chain  of  effectiveness  prediction  at  the  present  time. 

13.  Improved  techniques  are  required  for  conversion  of  available  data  to 
vector  and  matrix  elements  of  A,  D,  C, 

14.  Present  data  collection  and  analysis  on  operational  weapon  systems  is 
inadequate.  This  problem  should  be  partially  solved  by  implementation 
of  Task  Group  III  recommendations. 

15.  Additional  research  is  required  on  the  simulation  approach  to  effective¬ 
ness  evaluation  for  complex  systems. 

16.  The  elements  and  terms  used  in  system  effectiveness  prediction  and 
evaluation  are  in  need  of  clear  definition  and  standardization. 
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SECTION  IV 
RECOMMENDATIONS 

1.  The  proposed  analytical  framework  and  system  effectiveness  concepts 
should  be  accepted  as  an  initial  working  base. 

2.  The  proposed  effectiveness  evaluation  techniques  and  task  analysis 
should  be  validated  on  one  or  more  AF  systems,  qovering  each  of  the 
four  (4)  phases  of  the  system  life  cycle.  It  is  recommended  that  sys- 
tem(s)  in  the  early  Operational  Phcise  be  chosen  for  the  validation 
exercise,  and  through  recourse  to  system  reports,  files  and  other 
docunientation,  the  evaluations  for  the  earlier  phases  be  synthesized. 
Although  it  will  be  difficult  to  maintain  objectivity  in  such  a  synthesis 
it  is  felt  that  the  obvious  time  advantage  and  the  continuity  provided 
justify  the  attempt. 

3.  Evaluation  techniques  should  be  modified  in  accordance  with  results 
obtained  in  (2)  above. 

4.  The  minimum  validation  program  indicated  should  be  completed  before 
implementing  effectiveness  evaluation  procedures  contractually. 

5.  A  study  of  procedures  for  test  and  demonstration  of  effectivenisss, 
similar  to  those  outlined  for  reliability  in  MIL-R-27070  and 
MIL-STD-781  should  be  initiated. 

6.  Further  study  is  recommended  on  the  entire  question  of  "confidence" 
(quantitative  or  qualitative)  in  results  of  effectiveness  evaluations, 
both  in  combining  the  confidence  regions  for  the  estimates  or  product 
functions  and  in  guiding  the  decision  process  when  no  qua.ntitative  con¬ 
fidence  statement  is  possible. 

7.  It  is  recommended  that  work  on  decision  algorithms  for  application  to 
incentive  fee  contracts  be  initiated.  Methods  for  establishing  incentive 
fee  versus  leve.T  of  effectiveness  as  well  as  the  associated  subject  of 
producer’s  and  consumer's  risk  should  be  explored  in  detail. 
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8.  The  preparation  and  adoption  of  standard  terminology  and  notation 
for  system  effectiveness  should  be  accomplished  at  an  early  date. 

The  following  recommendations  are  identified  with  certain  of  the  specific 

tasks  required  to  predict  effectiveness: 

9.  Mission  Definition 

It  is  recommended  that  the  effectiveness  requirement  specified  in  the 
SOR  be  accompanied  by  a  complete  description  of  conditions  under 
which  this  requirement  must  be  met.  It  should  be  noted  that  opera¬ 
tional  requirements  as  specified  in  the  SOR  must  be  translated  faith¬ 
fully  into  contractual  technical  program  objectives. 

10.  System  Description 

There  should  be  a  series  of  contractually  required,  controlled  docu¬ 
ments  completely  describing  the  system  at  each  point  in  time  from 
the  earliest  stages  of  design  to  fully  operational  hardware. 

a.  These  documents  should  have  a  USAF  approved  distribution  list 
and  should  be  furnished  to  all  cognizant  branches  and  agencies 
of  the  USAF. 

b.  The  documents  should  be  capable  of  rapid  update,  so  as  to  depict 
the  status  of  the  system  in  near  real  time. 

11.  Data  Acquisition 

a.  Due  to  the  wide  disparity  in  quoted  failure  rates  for  similar  com¬ 
ponents,  it  is  recommended  that  a  single  authoritative  source  for 
collection,  analysis,  validation,  and  dissemination  of  this  type  of 
information  be  established.  Steps  should  be  taken  immediately  to 
establish  the  regulations,  standards,  or  other  control  documents 
needed  to  assure  the  availability  of  component  failure  rate  data. 

b.  At  the  present  time,  there  is  no  recognized  source  for  mainten¬ 
ance  data  and  checkout  and  repair  times  by  equipment  or  function. 
It  is  recommended  that  such  a  source  be  established. 


14 


12.  Parameter  Estimation  Techniques 


The  problem  of  estimation  is  a  difficult  one  at  best.  The  application 
of  maximum  likelihood,  least  squares,  Bayesian  statistics  and  s^^quen- 
tial  analysis,  to  name  a  few,  are  only  cursorily  treated  in  the  Task 
Group  II  report.  Il  iS  recommended  that  the  USAF  support  a  program 
to  accomplish  at  least  the  following: 

a.  Survey  of  methods  of  estimation 

b.  Survey  of  cases  of  methodology  validation. 
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